Skip to content

configure C API CI aligned with runtime bindings - #360

Closed
ethanglaser wants to merge 12 commits into
rfsaliev/c-api-cifrom
dev/eglaser/c-api-ci
Closed

configure C API CI aligned with runtime bindings#360
ethanglaser wants to merge 12 commits into
rfsaliev/c-api-cifrom
dev/eglaser/c-api-ci

Conversation

@ethanglaser

Copy link
Copy Markdown
Member

No description provided.

ethanglaser and others added 9 commits June 16, 2026 14:21
This pull request enhances the flexibility and predictability of blocked
data structures by introducing explicit control over the number of
elements per block (`blocksize_elements`) in addition to the existing
byte-based blocking (`blocksize_bytes`). It also improves test coverage
to verify the new behavior and edge cases.

**Blocking parameter improvements:**

* Added an optional `blocksize_elements` field to the
`BlockingParameters` struct, allowing users to specify the number of
elements per block directly. If set, this takes precedence over
`blocksize_bytes` when determining block size.
(`include/svs/core/data/simple.h`)

**Testing and validation:**

* Added test to demonstrate the pitfalls of relying solely on
`blocksize_bytes` for memory prediction.
* Extended unit tests to cover scenarios where `blocksize_elements` is
set, including checks for correct block size selection and memory
consumption predictions.
(`tests/svs/core/data/block.cpp`)
Latest macos github actions image was bumped and no longer supports
specified clang versions. This is because we take latest:
https://github.com/intel/ScalableVectorSearch/blob/main/.github/workflows/build-macos.yaml#L35
and as far as I can tell there is not an easy way to track this via
dependabot. Alternative would be to pin it to a version but then it
could sit idle and forgotten.
This pull request refactors the `Blocked` class template to inherit from
its allocator type instead of storing it as a member, and updates the
associated test to use a struct-based allocator. This change simplifies
allocator handling and improves compatibility with standard allocator
patterns.

**Core class refactoring:**

* `Blocked` now inherits from its allocator type (`Alloc`) instead of
storing an `Alloc allocator_` member, which simplifies construction,
copying, and access to the allocator. The `get_allocator()` method now
returns `*this` (as an allocator), and constructors have been updated
accordingly.

**Test improvements:**

* The test for `Blocked` with an allocator has been updated to use a
struct-based allocator (`I`), which provides a `value_type` and integer
value for testing propagation and compatibility with the new
inheritance-based implementation.
- Introduced `svs_id_filter_interface`  to define filtering operations.
- Implemented `svs_index_search_topK` to support an optional ID filter
for search operations.
- Updated existing search functions to use the new filtered search
capabilities.
- Added a new source file `filtered_search.hpp` containing the logic for
filtered top-K search.
- Modified existing samples and tests to demonstrate and validate the
new filtering functionality.
- Marked the previous `svs_index_search` function as deprecated,
directing users to use `svs_index_search_topK` instead.
## Summary

Adds `get_memory_usage()` returning the total number of bytes a Vamana
index has **allocated** — graph storage + vector data + metadata — so an
integrator can accurately report and bound SVS memory consumption.

Both the static `VamanaIndex` and the dynamic `MutableVamanaIndex` are
covered, and the method is plumbed through the orchestrator layers
(`VamanaInterface` virtual → `VamanaImpl` override → `Vamana` /
`DynamicVamana`) so it is callable on `svs::Vamana` and
`svs::DynamicVamana`.

## Why

Integrators (e.g. memory-bounded module hosts) need to account for
memory SVS allocates via `mmap`/blocked allocators, which bypass the
host's `malloc` accounting. The existing `blocksize_bytes()` reports
only the initial block size and is neither an upper nor lower bound on
the real footprint. `get_memory_usage()` reports the true allocated
total across all blocks plus metadata.

## What

Accounting is **capacity-based** (the bytes the containers have
reserved, not just live elements) so that block over-allocation is
reflected:

| Component | Source |
|---|---|
| `graph_bytes` | `graph_.get_data().capacity() * element_size()` |
| `data_bytes` | `data_.capacity() * data_.element_size()` |
| `metadata_bytes` (dynamic only) | slot-status vector + entry-point
list + estimate of the external/internal ID translation maps |

A `VamanaMemoryUsage { graph_bytes, data_bytes, metadata_bytes, total()
}` struct and `get_memory_breakdown()` expose the per-component split;
`get_memory_usage()` returns `get_memory_breakdown().total()`.

Notes:
- A `detail::dataset_allocated_bytes()` helper uses capacity-based
accounting when the dataset exposes `capacity()` (flat/blocked
`SimpleData`), and falls back to live element count otherwise (e.g.
`SQDataset`). No public accessors or signatures were changed.
- The ID-translation map byte size is not directly queryable; it is
estimated from the entry count (accurate to within a few percent), with
a comment noting the approximation.

## Tests

New unit tests at both the core-index and orchestrator levels for the
static and dynamic indices:
- `tests/svs/index/vamana/index.cpp`,
`tests/svs/index/vamana/dynamic_index.cpp`
- `tests/svs/orchestrators/vamana.cpp`,
`tests/svs/orchestrators/dynamic_vamana.cpp`

Assertions: usage `> 0` for a built index, breakdown components sum to
the total, and `graph_bytes`/`data_bytes` are non-zero. `[managers]` and
the touched index tags pass with no regressions.
…n) (#354)

## Summary

Exposes memory accounting in the **C API** for the Valkey-search
integration:

- `svs_index_get_memory_usage(index, size_t* out_bytes, err)` — total
allocated bytes.
- `svs_index_get_memory_breakdown(index, svs_memory_breakdown_t* out,
err)` — `{graph_bytes, data_bytes, metadata_bytes}` component split.
~~- `svs_index_element_size(index, size_t* out_bytes, err)` — bytes per
stored vector.~~ (keep at data level)

All follow the existing C API conventions (out-param + `svs_error_h`,
`wrap_exceptions`), matching the Phase-A design in the memory-accounting
contract (intel-innersource #333).

## Layers

- **C API** (`bindings/c`): the three functions +
`svs_memory_breakdown_t` in `svs_c.h`; interface virtuals + concrete
overrides in `src/index.hpp`; impls in `src/svs_c.cpp`.
- **Core / orchestrator**: brings in `get_memory_breakdown()`
(`MemoryBreakdown` struct + capacity-based
`svs::data::detail::dataset_allocated_bytes` helper) on `VamanaIndex` /
`MutableVamanaIndex` and through the orchestrator, plus an
`element_size()` accessor parallel to `dimensions()`. This mirrors the
approved public PR #345 so the C API can build and test standalone; once
#345 lands on `dev/c-api`, this reduces to just the C API layer.

## Tests

`bindings/c/tests/c_api_index.cpp` (static) and
`c_api_dynamic_index.cpp` (dynamic): usage > 0, breakdown total ==
usage, `graph_bytes`/`data_bytes` > 0 (metadata > 0 for dynamic),
`element_size == sizeof(float) * dimensions`, and null-arg handling.
Both test cases pass (84 / 166 assertions).

Related: builds on #345; memory-accounting
contract in intel-innersource #333 / #326.
Bumps the pinned LTO prebuilt SVS library
(`bindings/cpp/CMakeLists.txt`) from
`svs-shared-library-lto-nightly-2026-05-21-1429` to
`svs-shared-library-lto-nightly-2026-07-21-127`, which includes
`get_memory_usage()` / `get_memory_breakdown()` (#345).

## Why

The `Build and unit tests for C++ runtime bindings (with static library,
ON)` job links the runtime bindings against this prebuilt lib. The
previously-pinned nightly predated #345, so it failed:

```
bindings/cpp/src/dynamic_vamana_index_impl.h:73: error: no member named 'get_memory_breakdown' in 'svs::DynamicVamana'
```

The new nightly was built from `main` (post-#345, via the private-repo
submodule bump intel-innersource#338) and ships the method — verified
the tarball's headers contain `get_memory_breakdown`. This should turn
that CI job green.

Follows the pattern of #311 (Update SVS_URL in binaries).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the C API build/packaging and CI to more closely mirror the runtime bindings setup, with explicit LVQ/LeanVec expectations and an integration-style consumer build to validate the exported CMake package and symbol surface.

Changes:

  • Add SVS_REQUIRE_LTO_ARCHIVE and switch LVQ/LeanVec prebuilt downloads to the v0.4.0 release artifacts, with a CMake-version guard for DOWNLOAD_EXTRACT_TIMESTAMP.
  • Tighten the exported C API target so pure-C consumers don’t inherit OpenMP::OpenMP_CXX or a cxx_std_20 requirement.
  • Add C API consumer/integration tests and update the GitHub Actions workflow to build/test/package in the manylinux container, plus stricter LVQ/LeanVec test expectations.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
bindings/cpp/CMakeLists.txt Adds LTO-archive requirement toggle and uses release tarballs with CMake-version-safe FetchContent args.
bindings/c/CMakeLists.txt Prevents leaking C++/OpenMP requirements to consumers; aligns LVQ/LeanVec prebuilt handling with runtime bindings.
bindings/c/tests/consumer/main.c New C-only consumer smoke test for the installed C API package.
bindings/c/tests/consumer/CMakeLists.txt Standalone consumer project using find_package(svs_c_api) and linking svs::svs_c_api.
bindings/c/tests/CMakeLists.txt Adds compile-time expectation flag for LVQ/LeanVec tests based on build config.
bindings/c/tests/c_api_test_utils.h Makes compressed-storage assertions configuration-aware; adds storage_usable().
bindings/c/tests/c_api_storage.cpp Requires SQ storage to always succeed (public build invariant).
bindings/c/tests/c_api_index.cpp Skips build/search for compressed storage when unusable; requires SQ always works.
bindings/c/samples/simple.c Falls back to simple storage when LeanVec/LVQ are unavailable.
bindings/c/samples/save_load.c Same fallback behavior for save/load sample.
bindings/c/samples/dynamic.c Same fallback behavior for dynamic sample.
.github/workflows/build-cpp-runtime-bindings.yml Passes REQUIRE_LTO_ARCHIVE into the runtime bindings container build.
.github/workflows/build-c-api-bindings.yml Reworks C API CI to container build + artifact packaging + integration tests.
.github/scripts/test-c-api-unit.sh Runs unit tests and executes samples in the container.
.github/scripts/test-c-api-bindings.sh Integration test: validates package contents, exported symbols, and consumer build/run.
.github/scripts/build-cpp-runtime-bindings.sh Propagates SVS_REQUIRE_LTO_ARCHIVE into the CMake configure.
.github/scripts/build-c-api-bindings.sh New script to configure/build/install/package C API bindings with LVQ/LeanVec toggles.

Comment thread bindings/c/tests/consumer/main.c Outdated
Comment thread bindings/c/samples/simple.c Outdated
Comment thread bindings/c/samples/save_load.c Outdated
Comment thread bindings/c/samples/dynamic.c Outdated

@rfsaliev rfsaliev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good except:
error handle can be reused, so constructions like:

svs_error_free(error);
error = svs_error_create();

should be removed

@rfsaliev rfsaliev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGreatFM

@mergify mergify Bot mentioned this pull request Aug 10, 2026
@ethanglaser

Copy link
Copy Markdown
Member Author

Thanks for reviews. With rebasing to main and C API branches and original C API CI branch there is a lot going on. Closing this in favor of #362 which adds these changes directly to C API branch

ethanglaser added a commit that referenced this pull request Aug 11, 2026
#360 reopened directly
to C API branch

---------

Co-authored-by: Rafik Saliev <rafik.f.saliev@intel.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants